Scaling effortlessly over multiple nodes is one of the defining properties of CockroachDB. By maintaining a strongly-consistent database state across a network of machines, the distributed system can provide reliability and availability, while transparently tolerating disk, machine, and even datacenter failures. But not everyone has access to a datacenter at their fingertips, so we recently began looking into what it would take to run CockroachDB on a Raspberry Pi, one of the go-to tools of the modern day computer tinkerer.
Running CockroachDB on a Raspberry Pi
Going into it, we knew we were in for a challenge. CockroachDB had only ever been tested on the x86 computer architecture, and we were already aware of a few hurdles we would need to jump in order to get it working on the ARM architecture sported by Raspberry Pis. On top of this, we had also never successfully gotten CockroachDB working on a 32-bit architecture, and were afraid of subtle differences between the memory address sizes coming back to haunt us. Still, we felt we were up to the task, and after a lot trial and error and a little bit of luck, we’re happy to say that we were able to run CockroachDB on a Raspberry Pi.
The rest of this post will walk through the steps of compiling and running CockroachDB on a Raspberry Pi. While we were able to get the database running, we can make no guarantees that it does not contain hidden bugs, that it won’t delete all of your data, or that it won’t set your Pi on fire, because we do not officially support the ARM family of architectures or 32-bit CPU architectures in general. As such, your mileage may vary.
If you run into any issues while following these steps, hit us up on GitHub by filing an issue.
Hardware/Software Specs
Below are the specs of the Raspberry Pi that we used to get CockroachDB up and running. Other models will probably work as well, but again, no guarantees.
Model: Raspberry Pi 1 Model B
Architecture: ARMv6 (32-bit)
CPU: 700 MHz single-core ARM1176JZF-S
Memory: 512 MB (shared with GPU)
Note: Models with 1 GB of RAM instead of 512 MB may be able to skip step #5, as memory constraints may no longer be an issue.
OS: Raspbian Jessie (May 2016)
Storage: 8 GB SD Card
Cockroach SHA: fd3b6cfe118e182bf3564da481774c0adff30851
Installation Steps
1. Download Golang
We are going to be compiling CockroachDB from source on the Raspberry Pi itself to avoid having to deal with cross-compilation. The reason for this is that while cross-compiling pure Go applications is a breeze, the task suddenly gets a lot trickier when the applications begin dealing with cgo. To avoid this, we’ll just compile CockroachDB directly on the Raspberry Pi.
Because we’re compiling from source on our Pi, our first step is to get the Go toolchain up and running. Download and install the ARMv6 archive, and then follow the installation instructions. Note that at the time of writing this, go version 1.7.x is required to compile CockroachDB.
After this step, you should have a working GOPATH.
2. Download CockroachDB’s source
This step is nice and simple, just run
go get -d github.com/cockroachdb/cockroach
in your GOPATH.
3. Modify constants that assume a 64-bit architecture
In github.com/cockroachdb/cockroach/storage/engine/rocksdb.go there are two identical lines which both say:
const maxLen = 0x7fffffff
These constants are used to specify the maximum length array that Go will allow. This maximum length is 2^31 because the length must fit inside a signed 32-bit integer. However, because we are working on a 32-bit architecture, and because the elements in these arrays are multiple bytes in size, these arrays will exceed the size of the memory address space. To address this, we can just reduce the constant values by replacing them with:
const maxLen = 0x7fffff
4. Modify RocksDB compiler flags
CockroachDB’s cgo package wrapping RocksDB uses a compiler flag that is not available to the PI’s version of g++. Luckily, it is safe to simply remove this flag.
In github.com/cockroachdb/c-rocksdb/cgo_flags.go replace the line
// #cgo CXXFLAGS: -std=c++11 -fno-omit-frame-pointer -momit-leaf-frame-pointer
with
// #cgo CXXFLAGS: -std=c++11 -fno-omit-frame-pointer
Thanks to @linuxerwang for pointing this out in #6091.
5. Increase available RAM
Compiling and linking Go programs (especially big ones like CockroachDB) can take up more memory than we have available by default on the Raspberry Pi, so we need to work around this. We use the two tricks listed below to give the Go compiler enough memory.
5a. Configure Memory Split
The Pi (Model B) comes with 512 MB of memory shared between the video subsystem and the main processor. To assist during the compilation, we adjust this memory split in favor of the main processor by using the `raspi-config` utility.
sudo raspi-config
This will open up a configuration tool. Go to `Advanced Options`, then `Memory Split`, and give the GPU 64 MB of memory instead of 128 MB. After this, go to `Ok`, `Finish`, and you will be prompted to reboot your Pi. Go ahead and reboot.
5b. Increase Raspberry Pi’s Swapfile Size
By default, the Raspbian distribution comes with a 100 MB swapfile. This is on the small side, so we increase this size to 1024 MB in the file /etc/dphys-swapfile:
sudo vim /etc/dphys-swapfile
The default value in Raspbian is:
CONF_SWAPSIZE=100
We need to change this to:
CONF_SWAPSIZE=1024
Then we need to stop and start the service that manages the swapfile on Rasbian:
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
When this is complete, we should be able to verify that the swap size has increased by running the command free -m.
Because swapping on flash-based memory is generally considered bad, you might want to reset this swapfile size after completing the compilation.
6. Compile using the standard system allocator
If we tried to compile CockroachDB now, we would run into a large number of warnings. By default, CockroachDB chooses to replace the standard glibc memory allocator with jemalloc, and these warnings are coming from our cockroachdb/c-jemalloc package. While the warnings may be safe to ignore, we can compile CockroachDB with the standard allocator instead using the command:
cd $GOPATH/src/github.com/cockroachdb/cockroach
make build TAGS='stdmalloc'
If all goes well, CockroachDB should successfully compile and we will be left with a `cockroach` binary. You now have a distributed database ready to run on your Raspberry Pi! We can now start a local cluster by following the steps listed here on our docs. If we’re feeling ambitious, we can even drop this binary onto a few more Raspberry Pis and create a Raspberry Pi cluster.
When you get your cluster up and running, let us know what you’re using it for. Hit us up in the comments or email us at projects@cockroachlabs.com. If you get stuck somewhere, feel free to reach out for help on our forum or on GitHub.
